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Abstract 

Drowsiness detection is a solution for identifying signs of fatigue or sleepiness in individuals. One of the key 
features of our model is that it can detect drowsiness at night as well using Mobile cameras (infrared sensors). 
The system captures infrared images of the person's face and analyzes the physiological and behavioral cues 
related to drowsiness. Infrared sensors allow for drowsiness detection in low-light conditions, making it 
particularly useful for night-time scenarios such as night driving. The system can trigger alerts or 
interventions if drowsiness is detected, helping to prevent accidents or mistakes. We will be using libraries 
like OpenCV, TensorFlow, CNN, and VGG19 features in our model. By combining the accessibility of Android 
devices with the advanced capabilities of the Deep Learning algorithm, drowsiness detection using infrared 
sensors has the potential to greatly improve the safety and productivity of individuals in their daily lives. 


Keywords: Deep-Learning; OpenCV; TensorFlow; Transfer learning; VGG 1. 


1. Introduction 

Drowsy driving poses a significant threat on roads 
worldwide, with countless accidents, injuries, and 
fatalities attributed to driver fatigue each year, as 
highlighted by the National Highway Traffic Safety 
Administration. In response, a crucial initiative has 
emerged: the development of a Drowsiness 
Detection and Alert System. This innovative project 
aims to utilize advanced technology, including 
artificial intelligence and sensors, to detect signs of 
drowsiness in drivers and provide timely alerts, 
thereby preventing potential accidents [5]. By 
leveraging real-time monitoring and analysis, this 
system holds promise in enhancing road safety and 
minimizing the devastating impact of drowsy 
driving incidents, marking a pivotal step towards a 
safer future for all road users.Drowsy driving is a 
major problem on roads worldwide. Numerous 
collisions stem from drivers experiencing fatigue or 
lapses in attention while operating vehicles. As per 


research conducted by the National Highway Traffic 
Safety Administration (NHTSA), approximately 
72,000 accidents, resulting in 44,000 injuries and 
800 fatalities annually, are attributed to drowsy 
driving solely in the United States. This project aims 
to develop a Drowsiness Detection and Alert 
System that can detect drowsy driving behavior and 
alert the driver to prevent accidents [6]. 
1.1. Fundamental 

The Drowsiness Detection and Alert System utilizes 
computer vision and Deep Learning principles. A 
mobile camera installed on the dashboard of a 
vehicle is utilized to observe the driver's facial 
expressions and movements. These visual inputs are 
then analyzed by a Convolutional Neural Network 
(CNN) to detect indicators of drowsiness, such as 
variations in eye openness. The CNN undergoes 
training using a dataset comprising images of 
drivers exhibiting both open and closed eyes to learn 
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patterns associated with drowsiness. Upon 
identifying drowsiness, the system activates an alert 
mechanism, which may include audible alarms, 
vibrations, or flashing lights. Overall, this system is 
crucial for preventing accidents caused by drowsy 
driving by promptly detecting and alerting drivers to 
take a break [7]. 

1.2. Objective 
Automatic reading aims to provide a consistent and 
unbiased evaluation of short answer scripts, 
ensuring that all scripts are assessed using the same 
criteria and standards, without human bias or 
subjectivity. Automated grading systems aim to 
save time and resources by automating the grading 
process, reducing the need for manual grading, and 
providing prompt feedback to students [8]. 
Automatic grading systems are designed to handle 
a large volume of short answer scripts efficiently, 
making them suitable for high-stakes assessments, 
large-scale exams, or assessments with a large 
number of students. Given an answer to a question 
from a selected set of questions, we aim to evaluate 
it and give a qualitative score. The subjective nature 
of an answer makes it difficult to grade it uniformly 
across many human graders [9]. In addition, human 
graders tend to unknowingly grade an answer with 
their own biases towards the subject matter 
presented. Manual grading has another major 
drawback, the time required to grade essays can be 
significantly high. 

1.3. Scope 
The scope of the "Drowsiness Detection and Alert 
System" project is to design and implement a 
system that can detect drowsinessin individuals 
and provide alerts to preventaccidents caused by 
drowsy driving or operating heavy machinery. The 
system will utilize a blend of sensors and advanced 
learning methods to scrutinize the user'sactions and 
recognize indications of drowsiness, including 
sluggish eye movements, yawning, and alterations 
in posture. Once drowsiness is detected, the system 
will alert the user in the form of an audible orvisual 
warning, such as a beep or flashing light.The 
project strives to offer a dependable and user- 
friendly resolution to the issue of drowsy driving, 
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aiming to prevent accidents resulting from driver 
fatigue.The system can be implemented in various 
settings, such as in personal vehicles, commercial 
trucks, andheavy machinery. The project seeks to 
investigate the possibilities of employing deep 
learning techniques and Android applications to 
create inventive solutions aimed atenhancing road 
safety and mitigating accidents [10]. 
2. Literature Survey 
The "Drowsiness Detection and Alert System" 
project utilizes deep learning, transfer learning, and 
OpenCV to detect signs of drowsiness like slow 
eye movement and yawning. By providing timely 
alerts through audible or visual warnings, it aims to 
prevent accidents caused by drowsy driving, 
offering a reliable solution to enhance road safety 
and mitigate the risks of driver fatigue. Machine 
learning algorithms have been extensively used for 
drowsiness detection. Ayush Goel and Jainav 
Sharma proposed a model using machine learning, 
their approach used libraries like OpenCV, sciPy, 
and image magiCk to extract the features from the 
user. 

2.1.Literature Review 

e Ayush Goel, Jainav Sharma, Sumit Jindal, 
“Driver Drowsiness Detection System” (2023) [1]. 
Driving while intoxicated is a severe issue that may 
result in collisions, injuries, and fatalities. Machine 
learning algorithms might be trained to more 
precisely anticipate when a motorist is at risk of 
falling asleep by examining vast quantities of data 
on driving behavior. Features used by them include 
the frequency of rubbing eyes and difficulty in 
keeping the head up or eyes open.The libraries used 
by them include OpenCV, NumPy, and SciPy, to 
process data from sensors, Pillow, and 
ImageMagick.They also used NLP libraries like 
NLTK and spaCy for driver’s speech signs.The 
model concentrates on momentum as a possible 
simple method for documenting the transient state. 


e Jaspreet Singh Bajaj,Dr. Naveen Kumar 
“System and Method for Driver Drowsiness 
Detection Using Behavioral and Sensor- 
BasedPhysiological Measures”(2023)[2]. They 
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have used data from the National Highway Traffic 
Safety Administration (NHTSA). Based on the 
assessment of the study, integrating non-intrusive 
behavioral —§ measures with sensor-based 
physiological measurements results in better 
outcomes and circumvents certain challenges. A 
suggested hybrid model assists in detecting driver 
sleepiness under all circumstances. For 
investigating the transition from an alert to a 
drowsy state, the driver's facial features are 
captured using a camera as a behavioral indicator, 
while a GSR sensor is employed as a physiological 
gauge. 

e Jagbeer Singh, Ritika Kanojia “Driver 
Drowsiness Detection System - An approach by 
Machine Learning Application” (2022) [3]. In this 
paper, the authors present a deep learning-based 
algorithm for driver fatigue detection, integrating 
facial and eye movement features trained on real- 
world driving data to achieve high accuracy. They 
also propose a driver identification system 
alongside the fatigue detection model. 
Acknowledging the limitations of relying solely on 
eye states, they incorporate facial expressions to 
enhance detection accuracy. While deep learning 
offers improved performance over traditional 
methods, the authors recognize potential 
implementation challenges for widespread 
adoption. Nonetheless, their study sheds light on 
the efficacy of deep learning in fatigue detection 
and underscores the importance of continued 
research to enhance road safety. 

e Aslam Uddin, Saydul Akbar Mura “4D: A 
Real-Time Driver Drowsiness Detection Using 
Deep Learning” (2023) [4] The proposal 
introduces a CNN (Convolutional Neural Network) 
framework designed to detect driver drowsiness in 
real-time by analyzing the condition of their eyes. 
The model achieved an accuracy of 97.4% on the 
MRL Eye dataset, outperforming other models. 
The model detects three levels of drowsiness - 
alert, drowsy, and very drowsy - based on eye 
closure and movement. The authors developed a 
real-time application using the trained model, 
which can process video input from a camera 
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mounted on the dashboard of a car and alert the 
driver if they are detected to be drowsy. The study 
highlights using deep learning and machine 
learning for accurate drowsiness detection, 
potentially preventing accidents and saving lives 
on the road. 

3. Implemented System/ Proposed System 

3.1. Overview 
A driver drowsiness detection system identifies 
driver fatigue, a significant factor in road accidents. 
The system uses a combination of sensors, 
cameras, and algorithms to monitor driver behavior 
and movements and detect signs of drowsiness. 
Some of the common signs of drowsiness that the 
system can detect include eye closure duration, 
blinking frequency, and head-pose changes. This 
technology uses image-processing techniques to 
analyze video feeds from cameras to detect facial 
features, head posture, and eye movements [11]. 
The system then uses machine learning algorithms 
to learn the patterns of drowsiness based on the 
data collected from the sensors.Machine learning 
algorithms are trained on driver data to recognize 
signs of drowsiness like yawning, drooping 
eyelids, and impaired driving. Once the system 
detects signs of drowsiness, it alerts the driver in 
real-time through audio and visual alerts. The alert 
can take the form of an alarm, vibration, or 
message displayed on the dashboard. The driver 
can then take appropriate actions, such as taking a 
break, drinking coffee, or switching drivers. Driver 
drowsiness detection systems have several social 
applications, including improving road safety, 
healthcare, transportation, insurance, and 
government regulations [12]. This can help reduce 
the number of accidents caused by drowsy driving 
and save lives on the road. In summary, driver 
drowsiness detection systems are crucial 
technologies for preventing accidents due to driver 
fatigue and enhancing road safety. 
3.1.1. Existing System Architecture 

The Existing System works as follows: - 

° The EfficientNetBO architecture is the 
backbone of a gated recurrent unit (GRU) recurrent 
neural network, combining CNN and RNN 
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functionalities. This fusion allows for 
comprehensive analysis of sequential image data, 
particularly useful for real-time drowsiness 
detection in drivers [13]. 


° Here, the EfficientNetBO architecture 
serves as the foundation for a gated recurrent unit 
(GRU) recurrent neural network, amalgamating 
CNN and RNN functionalities. 


e The GRU _ network is tasked with 
processing a dataset comprising 600 facial images 
of the driver. Its objective is to meticulously 


GRU 


600 frames 


International Research Journal on Advanced Engineering Hub (IRJAEH) 


e ISSN: 2584-2137 
Vol. 02 Issue: 05 May 2024 
Page No: 1361 - 1369 


https://irjaeh.com 
https://doi.org/10.47392/IRJAEH.2024.0188 


analyze these images, seeking to discern subtle yet 
critical features that serve as indicators of 
drowsiness, especially focusing on the crucial 
moments preceding potential incidents [14]. 


° Transfer learning was utilized in the GRU 
module, integrating the — EfficientNetBO 
architecture in Figure 2. 


e The GRU network processes 600 facial 
images of the driver, aiming to identify key 
features indicative of drowsiness within the final 
moments in Figure 1. 


DROWSY 0.8 
AWAKE 0.2 


Figure 1 Existing System Architecture 


3.1.2. Proposed System Architecture 


Date Preprocessing 
MRL_Eye dataset 
Batch_size = 8 
Input_size = (80,80,3) 


Transfer Learning 
Loading pretrained model 
VGG19 


Adding Custom layers 
Dense(256. activation='relu') 
Dropout(0.5) 
Dense(2, activation='softmax’') 


Freeze pre-trained layers 
layertrainable = False 


Model is Ready to 
Predict 


Evaluation of Model 


Data Augmentation 
Train Data 
Validation Data 
Test Data 


Compiling model 
Optimizer = Adam 


Figure 2 Implemented System’s Model Training Workflow 
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e The Training dataset was further 
subdivided into two classes Open Eye and Close 
Eye to train the model similarly test data contained 
the same work to test the model for prediction. 

e CNN’s Transfer Learning methods VGG 
19 model is used to train the model on a given 
dataset and with appropriate fine-tuning of 
hyperparameters is done to make the model work 
accurately and in a precise way. 


Real time Face detection for 
drowsiness using webcam 


Capturing real time facial 
expression to detect 
drowsiness 


Sending Images as input to 
the system for prediction 


Yes 
Alert the driver usimg Alarm to 
Stay awake 


Figure 3 Real-Time Drowsiness Detection 

e The system operates by opening a mobile 
application where the system’s camera starts 
locating the ROI, that is, face, to detect drowsiness. 

e It starts by capturing facial expressions to 
detect drowsiness [15]. 

e It sends the [Real-Time Input to the system 
for detection 
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° When the system identifies signs of 
drowsiness in the driver, it raises the threshold 
score and notifies the driver with an alarm to 
remain alert. If not, it will continue to be evaluated 
until drowsiness is detected in Figure 3. 

3.2. Implementation Details 

3.2.1. Methodology 
The methodology employed in this study begins 
with the preprocessing of the MRL Eye Dataset, 
categorizing images into "Close Eyes" and "Open 
Eyes" for classification. Transfer learning is then 
utilized, leveraging the VGG19 pre-trained model, 
known for its robustness in image classification 
tasks. Custom layers are incorporated onto the pre- 
trained model to customize it for the particular 
purpose of identifying closed and open eyes. Pre- 
trained layers are frozen to retain learned features, 
and the model is compiled using appropriate 
optimization techniques and loss functions. Data 
augmentation methods are utilized to enhance the 
variety within the training dataset. The model is 
trained over multiple epochs, with real-time 
predictions integrated into the system for 
drowsiness and yawning detection using OpenCV 
and dlib libraries. Real-time visualization of 
drowsiness and yawning predictions provides 
immediate feedback to users, enhancing safety and 
awareness. 

3.2.2. Algorithm name 

e Transfer Learning with VGGI19: 
Transfer learning is applied using the VGG19 pre- 
trained convolutional neural network (CNN). 
VGGI19 is well-suited for image classification 
tasks due to its deep architecture and superior 
performance on large datasets like ‘imagenet’. It 
provides a solid foundation for feature 
extraction, enabling effective adaptation to the 
drowsiness detection task. 

e Face and Eye Detection with OpenCV: 
OpenCV and dlib libraries are employed for face 
and eye detection in real-time video streams. 
OpenCV__ offers robust computer vision 
functionalities, while dlib provides efficient facial 
landmark detection. This combination enables 
accurate localization of facial features, crucial for 
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subsequent drowsiness and yawning detection. 

e Custom Neural Network Layers: 
Specialized neural network layers are appended 
onto the pre-existing VGG19 model 
to tailor it for the specific objective of 
detecting drowsiness. These layers include 
Global Average Pooling 2D (GAP) for spatial 
dimension reduction, Dense layers for capturing 
complex patterns, and Dropout layers for 
regularization. Together, they enhance the model's 
ability to discern features indicative of drowsiness 
from input eye images. 

e Yawning Detection using Lip Landmark 
Analysis: Yawning detection is facilitated by 
analyzing the distance between upper and lower lip 
landmarks detected using dlib. By setting a 
predefined threshold for lip distance, yawning 
events can be accurately identified in real-time 
video frames. This approach provides an additional 
layer of drowsiness detection, complementing the 
eye-based detection system. 

3.2.3. Hardware and 
Specifications 
Table 1 Hardware Details 


Specification 


Software 


High resolution camera capable of capturing the driver's face in real-time 


Table 2 Software Details 


Component Specification 


OpenCV Library | A popular computer vision library used for face detection and other image 

processing tasks 

Dhib Library A machine learning library used for facial landmark detection and feature 
extraction 


Python Language A popular programming language used for developing the software 


components of the system 
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4. Result and Discussion 
4.1. Performance Evaluation Images 


Open Score:0 §) 


Closed Score:25 


Figure 4 Performance Evaluation Images 7 


4.2. Standard Dataset Used 
Table 3 Standard Dataset Used 


The dataset comprises infrared images captured 
under various lighting conditions and using 
different devices, encompassing both low and high 
resolutions in Figure 4. This dataset is appropriate 
for evaluating multiple features or trainable 
classifiers. To facilitate algorithm comparison, the 
images are categorized, making them suitable for 
training and testing classifiers. It contains 80 
thousand images of 37 people consisting of 33 men 
and 4 women in Table 1,2&3. 
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4.3. Evaluation Parameters 

Assessing the performance of a Drowsiness 
Detection and Alert System involves several key 
evaluation metrics: 

Accuracy: This metric evaluates the accuracy of 
drowsiness detection by determining — the 
proportion of accurately identified drowsy and 
awake states relative to the total sample size in 
Figure 5. 


Training Accuracy Over Epochs 


0.86 7 —@— Training Accuracy 


0.84 4 


0.82 4 


Accuracy 


0.80 5 


0.78 5 


T ; : : : 
74 4 6 8 10 
Epochs 


Figure 5 Accuracy 


Precision: Precision assesses the system's capacity 
to reduce false alarms by calculating the proportion 
of accurately identified drowsy instances among all 
positive predictions in Figure 6. 


1.0 


0.8 


Precision 


0.2 


0.0 
0.0 0.2 0.4 0.6 0.8 1.0 
Recall 


Figure 6 Precision 


Recall (Sensitivity): Recall, alternatively referred 
to as sensitivity or the true positive rate, evaluates 
the system's accuracy in detecting drowsy states by 
determining the proportion of true positives among 
all actual drowsy states. 
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F1 Score: The Fl score offers a well-rounded 
evaluation of the system's effectiveness by 
computing the harmonic mean of precision and 
recall. This approach accounts for both false 
positives and false negatives, providing a balanced 
perspective on performance in Figure 7. 


0.6 4 


Fl Score 
° 
u 
a 


r t r r r t r r r 
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 
Threshold 


Figure 7 F1 Score 


Confusion Matrix: A confusion matrix acts as a 
thorough instrument, distinguishing between true 
positives, true negatives, false positives, and false 
negatives, thereby providing a _ detailed 
understanding of the system's effectiveness. By 
analyzing these components, it enhances 
understanding of the model's precision and 
identifies areas for enhancement, empowering 
informed decision-making across applications like 
machine learning and predictive analytics in Figure 
8. 


3.0 


Test Confusion Matrix 


Close 


True label 


& > 
ro 
Predicted label 


Figure 8 Confusion Matrix 
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Conclusion 
By combining the accessibility of Android devices 
with the advanced capabilities of deep Learning 
algorithms, drowsiness detection using infrared 
sensors has the potential to greatly improve the 
safety and productivity of individuals in their daily 
lives. One of the key features of our model is that 
it can detect drowsiness at night as well using 
Mobile cameras (infrared sensors). The system 
uses the MRL Eye Dataset, a collection of images 
of the driver’ eyes, to identify signs of drowsiness. 
The EAR and MAR sensors monitor eye and 
mouth movements respectively and send an alert if 
detected drowsy on a mobile app to alert the driver. 
Through the analysis of these data streams, the 
system can detect the degree of drowsiness and 
notify the driver through either an alarm or 
vibration. Our drowsiness detection and alert 
system, trained with inception V3 and VGG19 
models, marks a_ substantial advancement in 
improving road safety. Achieving an accuracy of 
50% in inception V3 and 82% with the VGG19 
model is a commendable feat, and we have 
implemented a robust backend using Flask to 
deploy and run these models on the web. 
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